Collaborative search results

ABSTRACT

Methods, systems, and apparatus, including computer program products, for providing alternative search results for a query. In one aspect, a method includes transmitting a set of one or more search results for a query to a client device for presentation to a user, where each search result refers to a respective resource, receiving from the client device an alternative search result submitted by the user for the query, associating the alternative search result with the query, and storing in a repository the query and the alternative search result, where the alternative search result is transmitted with the set of one or more search results for a new search of the query.

BACKGROUND

This specification relates to providing alternative search results.

Internet and intranet search engines aim to identify resources (e.g.,web pages, images, documents, processes, multimedia content) that arerelevant to a user's needs and to present information about theresources in a manner that is most useful to the user. In response to aquery submitted by a user, search engines return search resultsreferring to resources identified as relevant to or matching the query.

Unfortunately, the search results returned by the search engine may notalign well with what the user has in mind, for example, if there isambiguity in the meaning of the query terms. Even if the search resultsreturned are relevant objectively to the query, the results may not berelevant to the user's actual subjective needs. Intranet search engines,in particular, may return few results relevant to a given query, due tothe small number of interlinked resources of an intranet relative to thelarge number of interlinked resources available on the Internet.

In some circumstances, the user may know of a resource that bettermatches the user's needs than the resources referred to by the returnedsearch results. Alternatively, the user may identify a resource thatbetter matches the user's needs after searching through a large numberof returned search results or through resources linked to a resourcereferred to by one of the returned search results. In these cases, theuser may want to share knowledge of the good resource with other userswho search using the same query or a similar query in the future.

Some search engines allow users to suggest resource addresses for thesearch engine to search and index. Some administrators of search systemswill accept user-suggested alternative search results, for example,through e-mails or an issue tracking system. Generally, an administratormust manually add the alternative search result (e.g., with customsoftware) to search results generated by the search engine before thealternative search result is available for presentation to other users.

SUMMARY

In general, one aspect of the subject matter described in thisspecification can be embodied in methods that include the actions oftransmitting a set of one or more search results for a query to a clientdevice for presentation to a user, where each search result refers to arespective resource, receiving from the client device an alternativesearch result submitted by the user for the query, associating thealternative search result with the query, and storing in a repositorythe query and the alternative search result, where the alternativesearch result is transmitted with the set of one or more search resultsfor a new search of the query. Other embodiments of this aspect includecorresponding systems and computer program products. The system includesa client device and one or more computers operable to interact with theclient device and to perform the above listed actions.

These and other embodiments can optionally include one or more of thefollowing features. Instructions and the alternative search result canbe transmitted to the client device for presentation to the user, wherethe instructions include instructions for displaying the alternativesearch result in a more prominent position than the positions of the oneor more search results in the set. In response to a new search of thequery submitted by a different user, instructions and the alternativesearch result can be transmitted to a different client device forpresentation to the different user, where the instructions includeinstructions for presenting the alternative search result and a userinterface for editing or deleting the alternative search result. Anidentifier for the user can be received from the client device, theidentifier for the user can be associated with the alternative searchresult, and the identifier for the user can be transmitted with thealternative search result so as to indicate that the alternative searchresult was submitted by the user. The query can include one or moreterms, and the alternative search result can be transmitted to a clientdevice in response to a search for a different query that includes atleast one of the one or more terms. A resource can be one or more of aweb page, an image, a document, a process, or multimedia content, andeach search result can include a title and a link to the respectiveresource. For each of a plurality of alternative search resultsassociated with a particular query, a number of times the alternativesearch result is presented to a user can be tracked. Alternatively or inaddition, for each of the plurality of alternative search resultsassociated with the particular query, a number of times the alternativesearch result is selected by a user can be tracked. The alternativesearch results in the plurality of alternative search results can beranked based on the tracked numbers. For the system, the one or morecomputers can include a server operable to interact with the clientdevice through a data communication network, and the client device canbe operable to interact with the server as a client. The client devicecan include a personal computer running a web browser or a mobiletelephone running a wireless application protocol (WAP) browser. The oneor more computers can include one personal computer, and the personalcomputer can include the client device.

One possible method includes receiving a set of one or more searchresults for a query, where each search result refers to a respectiveresource, causing the display of the set of one or more search resultsfor the query, receiving from a user an alternative search result forthe query, and providing the alternative search result to a searchengine, so that the alternative search result is available to users whosubmit new searches of the query. Other embodiments of this aspect ofthe subject matter described in this specification include computerprogram products.

These and other embodiments can optionally include one or more of thefollowing features. A user interface can be provided for receiving analternative search result for the query. The alternative search resultcan include a title and location for a resource. The display of thealternative search result and the set of one or more search results canbe caused. A user interface for editing or deleting the displayedalternative search result can be provided. An identifier for the usercan be provided to the search engine, and the identifier of the user canbe caused to be displayed in proximity to the alternative search resultso as to indicate that the alternative search result was submitted bythe user. The alternative search result can be displayed in a moreprominent position than the positions of the one or more search resultsin the set.

Particular embodiments of the subject matter described in thisspecification can be implemented to realize one or more of the followingadvantages. A user can add an alternative search result to the resultspage returned for a given query. The alternative search result isinstantaneously available and automatically returned with other searchresults when the same or a different user submits the given query. Thealternative search result can also be returned when a user submits adifferent query that is a synonym of the given query or that includes atleast one of the terms of the given query. A user-friendly interfaceprovides a way for users to directly submit, delete, or edit alternativesearch results which are immediately available to other users of asearch system without requiring administrator approval. The ease ofsubmission and the instantaneous availability encourage users to submitand collaborate on the alternative search results. This collaborativeeffort can produce search results which are more relevant to a givenquery than system-generated search results. The number of impressionsand the number of clicks of an alternative search result can be recordedand used in ranking multiple alternative search results for a givenquery. User identifiers can be recorded and displayed along with therespective alternative search results to provide accountability andpublic recognition of users who add good results.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example search system.

FIG. 2 shows an example process for providing an alternative searchresult.

FIG. 3 illustrates an example web page of search results including analternative search result.

FIG. 4 illustrates an example web page of search results including auser interface for submitting an alternative search result.

FIG. 5 illustrates an example web page of search results including analternative search result being submitted using the user interface ofFIG. 4.

FIG. 6 illustrates an example web page of search results including thealternative search result from FIG. 5.

FIG. 7 shows an example process for adding a new alternative searchresult for a query in an alternative search result repository.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 shows an example search system 1014 for providing search resultsrelevant to submitted search queries as can be implemented in aninternet, an intranet, or another client and server environment. Thesearch system 1014 is an example of an information retrieval system inwhich the systems, components, and techniques described below can beimplemented.

A user 1002 can interact with the search system 1014 through a clientdevice 1004 (e.g., a personal computer, a mobile telephone, a personaldigital assistant, a mobile audio or video player, a game console, or acombination of one or more of them). For example, the client 1004 can bea computer coupled to the search system 1014 through a local areanetwork (LAN), e.g., an enterprise intranet, or a wide area network(WAN), e.g., the Internet. In some implementations, the search system1014 and the client device 1004 can be one machine. For example, a usercan install a desktop search application on the client device 1004. Theclient device 1004 will generally include a random access memory (RAM)1006 and a processor 1008.

A user 1002 can connect to a search engine 1030 within a search system1014 to submit a query 1010. When the user 1002 submits a query 1010,the query 1010 is transmitted through one or more wired or wirelessnetworks to the search system 1014. The search system 1014 can beimplemented as, for example, computer programs running on one or morecomputers in one or more locations that are coupled to each otherthrough a network. The search system 1014 includes an index database1022 and a search engine 1030. The search system 1014 responds to thequery 1010 by generating search results 1028, which are transmittedthrough the network to the client device 1004 in a form that can bepresented to the user 1002 (e.g., as a search results web page to bedisplayed in a web browser running on the client device 1004).

When the query 1010 is received by the search engine 1030, the searchengine 1030 identifies relevant resources. The search engine 1030 willgenerally include an indexing engine 1020 that actively searches acorpus of resources (e.g., web pages, images, or news articles on anintranet or the Internet) to index the resources found in that corpus,and stores index information for the resources in an index database1022. This index database 1022 is used to identify resources that matchthe query 1010.

The search engine 1030 will generally include a ranking engine 1052 (orother software) to rank the resources related to the user query 1010.The ranking of the resources can be performed using conventionaltechniques for determining an information retrieval score for indexedresources in view of a given query. The relevance of a particularresource with respect to a particular query term or to other providedinformation may be determined by any appropriate technique. The searchengine 1030 can transmit the search results 1028 through the network tothe client device 1004 for presentation to the user 1002.

FIG. 2 shows an example process 2000 for providing an alternative searchresult. For convenience, the example process 2000 will be described withreference to FIGS. 3-6 and a client device (e.g., client device 1004 ofFIG. 1) that performs the process 2000.

The client device 1004 receives a set of one or more search results fora query, where each search result refers to a respective resource (step2010). The client device 1004 can receive the set of search results inresponse to user submission of the query using the client device 1004. Aresource can be one or more of a web page, an image, a document, aprocess, or multimedia content. Other resources are possible. A searchresult can include a title for the resource, a selectable link to theresource, and a snippet of content from the resource. The client device1004 causes the display of the set of one or more search results for thequery (step 2020). The search results can be displayed by a web browseron a display device that is part of or coupled to the client device1004.

FIG. 3 illustrates an example web page 3000 of search results 3005including an alternative search result 3020 a. The web page 3000includes a query 3010, “shuttle,” and a set of search results 3005generated and returned from a search system (e.g., the search engine1030 of the search system 1014 of FIG. 1) in response to the query 3010.In the example of FIG. 3, the system-generated search results 3005 aredirected to wireless network access on a shuttle. Even if the searchresults 3005 are relevant objectively to the query 3010, the searchresults 3005 may not be relevant to the user's actual subjective needs.For example, the user may be searching for a schedule of the shuttleservice, which cannot be accessed by selecting any of the search results3005 directed to wireless network access on the shuttle.

The search system 1014 can provide to the user one or more alternativesearch results for the query 3010. An alternative search result is asearch result that is submitted by a user of the search system 1014 tobe provided to other users as an alternative to search results (e.g.,search results 3005) generated by the search system 1014 for aparticular query. In some implementations, in response to receiving aquery submitted by a user, the search system 1014 retrieves, inparallel, the system-generated search results and the user-submittedalternative search results. For example, the search system 1014 canretrieve system-generated search results from a search engine index(e.g., index database 1022 of FIG. 1) while retrieving, in parallel,user-submitted alternative search results from an alternative searchresult repository. The search system 1014 can merge the retrieved searchresults and transmit the merged search results to the client device 1004for presentation to the user.

The search system 1014 can transmit to the client device 1004instructions for presenting the alternative search results to the user.These instructions can be provided with instructions for presenting thesystem-generated search results 3005. In various implementations, theinstructions can include one or more client-side scripts (e.g., usingJavaScript) which can be embedded within a hypertext document (e.g.,example web page 3000) and executed by a web browser running on theclient device 1004. The client-side scripts can associate regions in thehypertext document with user interface elements that, when selected,trigger one or more actions. A client-side script can includeinstructions for performing the one or more actions triggered by aselection.

The example web page 3000 includes a user-submitted alternative searchresult 3020 a. The alternative search result 3020 a includes a captionor title 3022 a and a location or address 3024 a for a resource. Thealternative search result 3020 a can be presented in proximity to (e.g.,above) the system-generated search results 3005 in the web page 3000. Insome implementations, the alternative search result 3020 a is displayedin a more prominent position of the web page 3000 (e.g., at the top ofthe web page 3000) than the positions of the system-generated searchresults 3005. The presentation instructions can include instructions forpresenting the alternative search result 3020 a in a way thatdistinguishes the alternative search result 3020 a from thesystem-generated search results 3005. For example, the alternativesearch result 3020 a can be displayed in a different color or font or ina shaded region of the web page 3000 that is separated from thesystem-generated search results 3005.

In some implementations, the client device 1004 provides to the searchengine 1030 of the search system 1014 an identifier 3030 a for the userwho submitted the alternative search result 3020 a. The web browser candisplay the identifier 3030 a in proximity to the alternative searchresult 3020 a, to indicate that the alternative search result 3020 a wassubmitted by that particular user. Displaying a user identifier 3030 awith an alternative search result 3020 a provides some accountabilityand promotes trust among users of the search system 1014, whiledeterring spam that might occur if alternative search results weresubmitted anonymously. In some implementations, the user identifier 3030a is a network address or a user login name, e.g., for an enterpriseintranet or for a public Internet search service. In someimplementations, the user identifier 3030 a is anchor text for ahyperlink to additional information about the user identified by theidentifier 3030 a.

In some implementations, a client-side script executed by the webbrowser includes instructions for displaying a user interface element3040 a (i.e., the “Edit” link) for editing the associated alternativesearch result 3020 a. Selection of the user interface element 3040 a cantrigger the web browser to display a user interface with the title 3022a and the address 3024 a of the alternative search result 3020 adisplayed in editable text boxes. The user can modify the title 3022 ato more accurately describe the resource or can update the address 3024a if it is outdated, for example. In effect, the ability of one user toedit an alternative search result submitted by another user provides amechanism for users to collaborate on providing relevant alternativesearch results. In some implementations, a user interface element (notshown) is provided to allow the user to delete the alternative searchresult 3020 a.

Returning to the example process 2000, the client device 1004 receivesfrom the user an alternative search result for the query (step 2030). Insome implementations, the client device 1004 provides a user interfacefor receiving an alternative search result for the query. In the exampleweb page 3000, if the user does not find the alternative search result3020 a or the system-generated search results 3005 useful, the user canadd another alternative search result for the query 3010 by selectingthe user interface element 3050 (i.e., the “Add another result” link).

In some scenarios, a user submits an alternative search result that ispart of the set of system-generated search results. For example, theuser may submit the alternative search result without knowing that it isincluded in the system-generated search results, e.g., because it is alater-listed result in a long list of results. Alternatively, the usermay know that the alternative search result is included in thesystem-generated search results but may submit the alternative searchresult regardless, because the user believes the result should bedisplayed in a more prominent position, e.g., at the top of the resultpage. In some implementations, the web browser can display a userinterface element (e.g., a selectable “promote to top” link) next toeach system-generated search result to allow a user to promote a resultto the top of the result page without needing to duplicate the result asan alternative search result.

FIG. 4 illustrates an example web page 4000 of search results 3005including a user interface 4050 for submitting an alternative searchresult. User selection of the user interface element 3050 (i.e., the“Add another result” link) of FIG. 3 can trigger the web browser todisplay the user interface 4050. The user interface 4050 includes atitle text box 4052 for entering a title for a resource and an addresstext box 4054 for entering an address for the resource (e.g., a locationwhere the resource content is stored or accessible). The resourceaddress can be, for example, a Uniform Resource Locator (URL), anInternet Protocol (IP) address, or a location in a network file system.In some implementations, the user can enter the resource title in thetitle text box 4052 as a character string with a placeholder for thequery (e.g., “<placeholder> schedule,” where <placeholder> would bereplaced with the query 3010, “shuttle”). The user interface 4050 caninclude a “Save” button 4060 for saving the alternative search resultand a “Cancel” button 4070 for canceling entry of the alternative searchresult. In some implementations, the “Save” button 4060 is notselectable until the user has entered an alternative search result inthe user interface 4050.

In some implementations, the user interface 4050 includes different oradditional user interface elements, e.g., elements that allow the userto specify how an alternative search result should be matched with oneor more queries. For example, the user interface 4050 can include abutton or a link to additionally submit an alternative search resultentered for one query (e.g., “shuttle”) for another query (e.g., “bus”),so that the alternative search result will be available to users whosubmit either queries. As another example, the user interface 4050 couldinclude a text field that allows the user to list all the queries (e.g.,“shuttle,” “bus,” “shuttle bus,” and “transport”) for which analternative search result is to be matched, so that the alternativesearch result will be available to users who submit any of the listedqueries. In some implementations, an alternative search result for agiven query will be returned for a search of any query that includes atleast one of the terms of the given query, as will be described in moredetail below.

FIG. 5 illustrates an example web page 5000 of search results 3005including an alternative search result being submitted using the userinterface 4050 of FIG. 4. In this example, the user has entered a title3022 b, “Shuttle Schedule,” into the text box 4052 and an address 3024b, “www.yourlocalintranetsite.com/shuttle/schedule.html,” into the textbox 4054 for the new alternative search result. Once the user issatisfied with the new alternative search result entered in the userinterface 4050, the user can select the “Save” button 4060.

Returning to the example process 2000, the client device 1004 providesthe new alternative search result to a search engine (e.g., searchengine 1030 of search system 1014), so that the new alternative searchresult is available to users who submit new searches of the query 3010(step 2040). User selection of the “Save” button 4060 can trigger theweb browser, executing a client-side script, to send a request to thesearch system 1014 to add the alternative search result. In someimplementations, the client device 1004 causes the display (e.g.,automatically) of the new alternative search result with the set of oneor more system-generated search results 3005 and any existingalternative search results (e.g., result 3020 a) for the query 3010. Forexample, once the user submits the new alternative search result, it canbe available substantially immediately to all users of the search system1014 without the new alternative search result needing to be manuallyadded by the administrator of the search system 1014.

FIG. 6 illustrates an example web page 6000 of search results 3005including the alternative search result from FIG. 5. The new alternativesearch result submitted by the user using the user interface 4050 ofFIG. 5 is displayed as alternative search result 3020 b below theexisting alternative search result 3020 a. In some implementations, thealternative search results 3020 a and 3020 b are displayed in a moreprominent position than the positions of the system-generated searchresults 3005. As an example, the alternative search results 3020 a and3020 b are displayed at the top of the web page 6000, above the set ofsystem-generated search results 3005.

The example web page 6000 includes an identifier 3030 b for the user whosubmitted the new alternative search result 3020 b. The identifier 3030b is displayed in proximity to the new alternative search result 3020 bso as to indicate that the new alternative search result 3020 b wassubmitted by that user. Displaying the user identifier providesaccountability while publicly rewarding users who submit goodalternative search results. The example web page 6000 also includes auser interface element 3040 b for editing the associated new alternativesearch result 3020 b. A different user who searches for the same query3010 will be presented with the multiple alternative search results 3020a and 3020 b and can edit or, optionally, delete either alternativesearch results 3020 a or 3020 b. The ability for users to edit or deletealternative search results submitted by other users provides a means forcommunity moderation. The different user can also add anotheralternative search result by selecting the user interface element 3050.

In some implementations, a user can provide additional content whensubmitting an alternative search result for a given query. For example,the user interface 4050 of FIG. 4 can provide additional text boxes foradding further information (e.g., comments) about the resource referredto by the alternative search result. In one example, if the resourceincludes information about a particular department or service (e.g., acredit union), additional text boxes could be used to add a location,working hours, or phone numbers. If the resource includes informationabout a user (e.g., an employee), a text box could be used to add anaddress for the user's personal web site. The user interface 4050 canalso include user interface elements to allow a user to upload content(e.g., an image of the user) to be displayed with the alternative searchresult or to provide an address for content (e.g., a location in anetwork file system where the user's résumé is stored) to be displayedas a hyperlink with the alternative search result.

In some implementations, a user can specify how the alternative searchresult will be available to users of the search system 1014. Forexample, if the user is providing an alternative search result for aquery about an upcoming event, the user may wish to limit the periodduring which the alternative search result will be returned for a searchof the query. The user interface 4050 could include checkboxes forvarious time periods (e.g., a week, a month, a quarter of a year) orcould provide a calendar feature for the user to select a start date andan end date to specify when the alternative search result would beavailable to users.

In some implementations, the set of alternative search results for aquery returned for a particular user may depend on one or moreparameters, e.g., the language of the search system interface used bythe user, the location of the user, or the membership of the user in aparticular group. In some implementations, a user submitting analternative search result can specify the subpopulation of users forwhich the alternative search result will be available. For example, ifthe user is providing an alternative search result that refers to aresource that is of interest to a particular department (e.g.,engineering, marketing, or legal) or user group, the user can specifythat the alternative search result will only be returned for users inthe particular group, e.g., by selecting a checkbox included in the userinterface 4050 for the particular group. In other implementations, thesubpopulation of users for which an alternative search result will beavailable can be automatically limited. For example, alternative searchresults submitted in German could be available only to users located inGermany or only to users who use a search system interface presented inGerman. In some implementations, a user submitting a query can specifyone or more parameters that affect the set of alternative search resultsreturned for the query. For example, if the user is submitting the queryin French but is fluent in German, the user can specify or request thatthe user receives alternative search results for German users. Somerequests can be rejected by a system administrator or automaticallydenied by the search system, for example, if the user is not authorizedto receive the requested alternative search results (e.g., alternativesearch results restricted to managers only).

FIG. 7 shows an example process 7000 for adding a new alternative searchresult for a query in an alternative search result repository. Forconvenience, the example process 7000 will be described with referenceto a system (e.g., the search system 1014 of FIG. 1) that performs theprocess 7000.

The system transmits a set of one or more search results for a query toa client device for presentation to a user, where each search resultrefers to a respective resource (step 7010). The system receives fromthe client device an alternative search result submitted by the user forthe query (step 7020). The alternative search result can be received asa request sent from the client device to add the alternative searchresult. The received alternative search result can include a title for aresource, an address for the resource, and other content related to theresource.

The system associates the alternative search result with the query (step7030), so that the alternative search result will be returned for anyuser that submits a new search of the query. In some implementations,the system also associates an alternative search result submitted for agiven query with other queries that are synonyms of the given query orthat include terms that are synonyms of the terms of the given query.

In some implementations, the system associates the alternative searchresult with each term of the one or more terms of a given query, so thatthe system returns the alternative search result in response to a searchfor any query that includes at least one of the one or more terms of thegiven query. For example, if the user submits an alternative searchresult 3020 b of FIG. 6 for a query “shuttle bus,” the search system1014 can associate the alternative search result 3020 b with the terms“shuttle” and “bus.” As a result, if a different user submits on aclient device a query for “shuttle service,” for example, the searchsystem 1014 can transmit the alternative search result 3020 b to theclient device for presentation to the different user.

In some scenarios, if a given query has multiple terms (e.g., “shuttlebus”), a result page for the given query might include duplicate orequivalent (e.g., different title and same address) alternative searchresults if one or more users submitted alternative search resultsreferring to the same resource for queries of different terms (e.g.,“shuttle” and “bus”) of the given query. In some implementations, thesystem can prevent duplicate or equivalent alternative search resultsfrom being returned for a query, for example, by comparing the addressesof the alternative search results associated with the query or the queryterms and disassociating from the query or the query terms all but oneof the alternative search results with identical or equivalentaddresses.

In some implementations, the system receives from the client device anidentifier for the user who submitted the alternative search result. Thesystem can associate the user identifier with the alternative searchresult and can transmit the user identifier with the alternative searchresult, so as to indicate that the alternative search result wassubmitted by the user identified by the identifier.

Returning to the example process 7000, the system stores in a repository(e.g., a database) the query and the alternative search result, wherethe alternative search result is transmitted with the set of one or moresearch results for a new search of the query (step 7040). Data in therepository can be accessed or processed using proprietary or open-sourcecomputer languages, e.g., Structured Query Language (SQL) or MySQL.

The system can transmit the alternative search result to a client devicefor presentation to a user who submits a search for the query. Thealternative search result can be retrieved from the repository and madeavailable to the user as dynamic content in a hypertext document (e.g.,a web page of search results) using server-side scripts. For example,the server-side scripts can be Hypertext Preprocessor (PHP) scriptsembedded within the result page and executed by a server in the searchsystem 1014 before the result page is sent to a client device 1004 forpresentation to the user.

In some implementations, the system transmits to the client deviceinstructions with the alternative search result. The instructions caninclude instructions (e.g., as a client-side script) for displaying thealternative search result in a more prominent position than thepositions of the one or more search results in the set. In someimplementations, in response to a new search of the query submitted by adifferent user, the system transmits the alternative search result withinstructions to a different client device for presentation to thedifferent user. The instructions can include instructions (e.g., as aclient-side script) for presenting the alternative search result and auser interface for editing or deleting the alternative search result.

In some scenarios, a particular query is associated with multiplealternative search results. In some implementations, for each of themultiple alternative search results, the system tracks the number oftimes the alternative search result is presented to a user (i.e., animpression), the number of times the alternative search result isselected (e.g., clicked) by a user, or both numbers. The system can rankthe multiple alternative search results based on the tracked numbers,and the multiple alternative search results can be ordered according torank when presented to a user. For example, the alternative searchresults can be ranked according to the number of times each result isselected by a user, where the alternative search result with thegreatest number of selections is displayed at the top of the list ofalternative search results. In another example, the alternative searchresults can be ranked according to a click through rate (e.g., the ratioof the number of selections to the number of impressions) computed foreach result, where the alternative search result with the highest clickthrough rate is displayed at the top of the list of alternative searchresults. In some implementations, the number of alternative searchresults presented to a user is limited to a predetermined number N, sothat only the top N ranked alternative search results are presented. Insome implementations, an alternative search result is available forpresentation to users during a predetermined trial period. In someimplementations, an alternative search result will be available beyondthe predetermined trial period if it exceeds a minimum threshold, e.g.,a selection threshold or a click through rate threshold.

In some implementations, in addition to queries and associatedalternative search results (e.g., as titles and addresses), the systemstores additional information in the repository. For a given alternativesearch result, the repository can include, for example, an identifierfor the user, the date and time of the submission, the number ofimpressions, the number of selections, the click through rate, orcontent, as described above. In some implementations, the system cangenerate automated messages based on the information stored in therepository to provide feedback to users who submit alternative searchresults. For example, the search system 1014 can send a message to auser (e.g., as an automatically generated e-mail addressed to the useridentifier) notifying the user that one of the user's submittedalternative search results has been edited or deleted. The message caninclude the modified alternative search result if it was edited and anidentifier for the user who edited or deleted the result, providing ameans for the users to collaborate on how the alternative search resultshould be specified. In one example, the user can be notified if analternative search result submitted by the user is being removed by thesearch system 1014, e.g., because the trial period is over or the rankis too low. In another example, the user can be notified if analternative search result submitted by the user is receiving a highnumber of selections or has a high click through rate, as a form ofpositive reinforcement.

In some implementations, the system maintains profile pages for userswho submit alternative search results. A user can access the user'sprofile page to view, for example, a history of the user's submittedalternative search results, including the number of impressions, thenumber of selections, and the click through rate for each result. Thisinformation can be retrieved from the repository and made available tothe user as dynamic content in a hypertext document (e.g., a web page)using server-side scripts. The profile page can additionally include aquality rating (e.g., stars) for the user or the user-submittedalternative search results as an incentive or a reward for users whosubmit results that are popular with other users. In someimplementations, a quality rating is displayed with a user identifierwhen the user identifier is displayed in proximity to an alternativesearch result. In some implementations, if the user has submitted aparticular alternative search result for multiple queries, the profilepage includes a tool that allows the user to update the address of theparticular alternative search result once for all the queries,eliminating the need for the user to independently update the resultaddress for each of the multiple queries.

In some implementations, the alternative search results submitted by auser are available to all users of the search system 1014 (e.g., acorporate intranet search system). In other implementations, thealternative search results submitted by a user are available to a subsetof the users of the search system 1014 (e.g., a public Internet searchsystem). For example, an alternative search result can be available to asmall subset of the user population for an initial trial period. If thealternative search result exceeds a minimum threshold (e.g., for anumber of selections or a click through rate), the search system 1014could make the alternative search result available to a larger subset ofthe user population, with the potential for the result to be madeavailable to the entire user population. This reduced impression settechnique could be used for a search system of a social network, where auser's alternative search result is initially available only to theuser's network of friends. As the alternative search result receivesadditional selections or achieves a higher click through rate, thealternative search result could be made available to a progressivelylarger network of friends and acquaintances.

In some implementations, the alternative search results are managed,stored, and served by a collaborative search system that is independentof the search engine-based search system (e.g., the search system 1014of FIG. 1). For example, a user can submit a query in an interface ofthe collaborative search system. In response, the collaborative searchsystem can simultaneously send the query to one or more searchengine-based search systems (e.g., systems selected by the user) andsearch an alternative search result repository on the collaborativesearch system. When system-generated search results are returned fromthe one or more search engine-based search systems, the collaborativesearch system can modify the returned system-generated search results toinsert the retrieved alternative search results before transmitting thecombined search results to the client device for presentation to theuser. The separate collaborative search system can provide thecollaborative result feature to users without requiring modifications tothe search engine-based search systems.

In some implementations, the collaborative result feature can beprovided by a web browser running on the client device. For example, theuser can use a custom web browser (or add a web browser plug-in), whichautomatically searches for alternative search results whenever the usersubmits a query in an interface of any search system. The search foralternative search results can be transparent to the user, who submitsthe query directly to the search system of the user's choice. The webbrowser can modify the result page returned by the chosen search systemto insert any retrieved alternative search results before displaying theresult page to the user.

In some implementations, alternative search results can be stored andsearched on a user's client device. For example, the user can use acustom web browser (or add a web browser plug-in), which stores a cacheof alternative search results on the client device. When the usersubmits a query, the cache can be searched to provide alternative searchresults for the query. The ability to retrieve alternative searchresults maintained locally on the client device reduces the need tosearch for alternative search results on the network and, hence, reducesnetwork traffic. The client device can periodically contact thecollaborative search engine to provide new alternative search resultssubmitted by the user and to retrieve new alternative search resultssubmitted by other users.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe subject matter described in this specification can be implemented asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a tangible program carrier forexecution by, or to control the operation of, data processing apparatus.The tangible program carrier can be a computer-readable medium. Thecomputer-readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal (e.g., amachine-generated electrical, optical, or electromagnetic signal), or acombination of one or more of them.

The term “data processing apparatus” encompasses all apparatus, devices,and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub-programs, orportions of code). A computer program can be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio or video player, a game console, a GlobalPositioning System (GPS) receiver, to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described is this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter described in thisspecification have been described. Other embodiments are within thescope of the following claims. For example, the actions recited in theclaims can be performed in a different order and still achieve desirableresults. As one example, the processes depicted in the accompanyingfigures do not necessarily require the particular order shown, orsequential order, to achieve desirable results. In certainimplementations, multitasking and parallel processing may beadvantageous.

What is claimed is: 1-18. (canceled)
 19. A computer-implemented method comprising: receiving a submission of a first resource locator for a first alternative search result by a first user using a first user device, wherein the alternative search result is associated with a first subpopulation of users of a search engine and with a first query; receiving a second query submitted by a different second user; determining that the first query matches the second query and that the different second user is a member of the first subpopulation of users and, in response, providing the alternative search result for presentation to the second user along with one or more other search results generated by the search engine in a response to the second query; receiving an indication of a selection, by the second user, of the alternative search result and, in response, incrementing a selection count representing a number of times that the alternative search result was selected by users in the first subpopulation of users to generate a modified selection count; determining that the modified selection count exceeds a threshold number and, in response, determining a second subpopulation of users to which the alternative search result will be provided; receiving a third query submitted by a different third user, the different third user not being a member of the first subpopulation of users; and determining that the first query matches the third query and that the different third user is a member of the second subpopulation of users and, in response, providing the alternative search result for presentation to the third user along with one or more other search results generated by the search engine in a response to the third query.
 20. The method of claim 19, wherein generating the second subpopulation of users comprises generating a subpopulation of users that includes the first subpopulation of users.
 21. The method of claim 19, further comprising generating the first subpopulation of users from a plurality of friends of the first user in a social networking system.
 22. The method of claim 19, wherein the second subpopulation of users comprises all users of the search engine.
 23. The method of claim 19, wherein determining that the modified selection count satisfies a threshold comprises determining that the modified selection count is greater than or equal to a particular threshold number of user selections.
 24. The method of claim 19, wherein determining that the modified selection count satisfies a threshold comprises determining that a click-through-rate based on the modified selection count satisfies a threshold.
 25. The method of claim 19, wherein the response to the second query or the response to the third query comprises a search results page that provides an identifier of the first user along with the first alternative search result submitted by the first user.
 26. The method of claim 19, wherein the first resource locator of the first alternative search result is not identified by any of a number of top-ranked search results previously provided to the second user or the third user.
 27. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving a submission of a first resource locator for a first alternative search result by a first user using a first user device, wherein the alternative search result is associated with a first subpopulation of users of a search engine and with a first query; receiving a second query submitted by a different second user; determining that the first query matches the second query and that the different second user is a member of the first subpopulation of users and, in response, providing the alternative search result for presentation to the second user along with one or more other search results generated by the search engine in a response to the second query; receiving an indication of a selection, by the second user, of the alternative search result and, in response, incrementing a selection count representing a number of times that the alternative search result was selected by users in the first subpopulation of users to generate a modified selection count; determining that the modified selection count exceeds a threshold number and, in response, determining a second subpopulation of users to which the alternative search result will be provided; receiving a third query submitted by a different third user, the different third user not being a member of the first subpopulation of users; and determining that the first query matches the third query and that the different third user is a member of the second subpopulation of users and, in response, providing the alternative search result for presentation to the third user along with one or more other search results generated by the search engine in a response to the third query.
 28. The system of claim 27, wherein generating the second subpopulation of users comprises generating a subpopulation of users that includes the first subpopulation of users.
 29. The system of claim 27, wherein the operations further comprise generating the first subpopulation of users from a plurality of friends of the first user in a social networking system.
 30. The system of claim 27, wherein the second subpopulation of users comprises all users of the search engine.
 31. The system of claim 27, wherein determining that the modified selection count satisfies a threshold comprises determining that the modified selection count is greater than or equal to a particular threshold number of user selections.
 32. The system of claim 27, wherein determining that the modified selection count satisfies a threshold comprises determining that a click-through-rate based on the modified selection count satisfies a threshold.
 33. The system of claim 27, wherein the response to the second query or the response to the third query comprises a search results page that provides an identifier of the first user along with the first alternative search result submitted by the first user.
 34. The system of claim 27, wherein the first resource locator of the first alternative search result is not identified by any of a number of top-ranked search results previously provided to the second user or the third user.
 35. A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving a submission of a first resource locator for a first alternative search result by a first user using a first user device, wherein the alternative search result is associated with a first subpopulation of users of a search engine and with a first query; receiving a second query submitted by a different second user; determining that the first query matches the second query and that the different second user is a member of the first subpopulation of users and, in response, providing the alternative search result for presentation to the second user along with one or more other search results generated by the search engine in a response to the second query; receiving an indication of a selection, by the second user, of the alternative search result and, in response, incrementing a selection count representing a number of times that the alternative search result was selected by users in the first subpopulation of users to generate a modified selection count; determining that the modified selection count exceeds a threshold number and, in response, determining a second subpopulation of users to which the alternative search result will be provided; receiving a third query submitted by a different third user, the different third user not being a member of the first subpopulation of users; and determining that the first query matches the third query and that the different third user is a member of the second subpopulation of users and, in response, providing the alternative search result for presentation to the third user along with one or more other search results generated by the search engine in a response to the third query.
 36. The computer program product of claim 35, wherein generating the second subpopulation of users comprises generating a subpopulation of users that includes the first subpopulation of users.
 37. The computer program product of claim 35, wherein the operations further comprise generating the first subpopulation of users from a plurality of friends of the first user in a social networking system.
 38. The computer program product of claim 35, wherein the second subpopulation of users comprises all users of the search engine. 